{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "(enable-model-monitoring)=\n",
    "# Enable model monitoring\n",
    "\n",
    "```{note}\n",
    "This is currently a beta feature.\n",
    "```\n",
    "\n",
    "To see tracking results, model monitoring needs to be enabled in each model.\n",
    "\n",
    "To utilize drift measurement, supply the train set in the training step.\n",
    "\n",
    "**In this section**\n",
    "- [Enabling model monitoring](#enabling-model-monitoring)\n",
    "- [Model monitoring demo](#model-monitoring-demo)\n",
    "  - [Deploy model servers](#deploy-model-servers)\n",
    "  - [Simulating requests](#simulating-requests)\n",
    "\n",
    "## Enabling model monitoring\n",
    "\n",
    "Model activities can be tracked into a real-time stream and time-series DB. The monitoring data\n",
    "is used to create real-time dashboards and track model accuracy and drift. \n",
    "To set the tracking stream options, specify the following function spec attributes:\n",
    "    \n",
    "   `fn.set_tracking(stream_path, batch, sample)`\n",
    "    \n",
    "- **stream_path**\n",
    "  - Enterprise: the v3io stream path (e.g. `v3io:///users/..`)\n",
    "  - CE: a valid Kafka stream (e.g. `kafka://kafka.default.svc.cluster.local:9092`)\n",
    "- **sample** &mdash; optional, sample every N requests\n",
    "- **batch** &mdash; optional, send micro-batches every N requests\n",
    "    \n",
    "## Model monitoring demo\n",
    "Use the following code to test and explore model monitoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Set project name\n",
    "project_name = \"demo-project\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy model servers\n",
    "Use the following code to deploy a model server in the Iguazio instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from sklearn.datasets import load_iris\n",
    "\n",
    "from mlrun import import_function, get_dataitem, get_or_create_project\n",
    "from mlrun.platforms import auto_mount\n",
    "\n",
    "project = get_or_create_project(project_name, context=\"./\")\n",
    "project.set_model_monitoring_credentials(os.environ.get(\"V3IO_ACCESS_KEY\"))\n",
    "\n",
    "# Download the pre-trained Iris model\n",
    "get_dataitem(\"https://s3.wasabisys.com/iguazio/models/iris/model.pkl\").download(\n",
    "    \"model.pkl\"\n",
    ")\n",
    "\n",
    "iris = load_iris()\n",
    "train_set = pd.DataFrame(\n",
    "    iris[\"data\"],\n",
    "    columns=[\"sepal_length_cm\", \"sepal_width_cm\", \"petal_length_cm\", \"petal_width_cm\"],\n",
    ")\n",
    "\n",
    "# Import the serving function from the Function Hub\n",
    "serving_fn = import_function(\"hub://v2_model_server\", project=project_name).apply(\n",
    "    auto_mount()\n",
    ")\n",
    "\n",
    "model_name = \"RandomForestClassifier\"\n",
    "\n",
    "# Log the model through the projects API so that it is available through the feature store API\n",
    "project.log_model(model_name, model_file=\"model.pkl\", training_set=train_set)\n",
    "\n",
    "# Add the model to the serving function's routing spec\n",
    "serving_fn.add_model(\n",
    "    model_name, model_path=f\"store://models/{project_name}/{model_name}:latest\"\n",
    ")\n",
    "\n",
    "# Enable model monitoring\n",
    "serving_fn.set_tracking()\n",
    "\n",
    "# Deploy the function\n",
    "serving_fn.deploy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simulating requests\n",
    "Use the following code to simulate production data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from time import sleep\n",
    "from random import choice, uniform\n",
    "\n",
    "iris_data = iris[\"data\"].tolist()\n",
    "\n",
    "while True:\n",
    "    data_point = choice(iris_data)\n",
    "    serving_fn.invoke(\n",
    "        f\"v2/models/{model_name}/infer\", json.dumps({\"inputs\": [data_point]})\n",
    "    )\n",
    "    sleep(uniform(0.2, 1.7))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
